Collaborative and Multi-Modal Mixed Reality for Enhancing Cultural Learning in Virtual Heritage

No abstrac

A comparison of immersive realities and interaction methods: cultural learning in virtual heritage

In recent years, Augmented Reality (AR), Virtual Reality (VR), Augmented Virtuality (AV), and Mixed Reality (MxR) have become popular immersive reality technologies for cultural knowledge dissemination in Virtual Heritage (VH). These technologies have been utilized for enriching museums with a personalized visiting experience and digital content tailored to the historical and cultural context of the museums and heritage sites. Various interaction methods, such as sensor-based, device-based, tangible, collaborative, multimodal, and hybrid interaction methods, have also been employed by these immersive reality technologies to enable interaction with the virtual environments. However, the utilization of these technologies and interaction methods isn’t often supported by a guideline that can assist Cultural Heritage Professionals (CHP) to predetermine their relevance to attain the intended objectives of the VH applications. In this regard, our paper attempts to compare the existing immersive reality technologies and interaction methods against their potential to enhance cultural learning in VH applications. To objectify the comparison, three factors have been borrowed from existing scholarly arguments in the Cultural Heritage (CH) domain. These factors are the technology’s or the interaction method’s potential and/or demonstrated capability to: (1) establish a contextual relationship between users, virtual content, and cultural context, (2) allow collaboration between users, and (3) enable engagement with the cultural context in the virtual environments and the virtual environment itself. Following the comparison, we have also proposed a specific integration of collaborative and multimodal interaction methods into a Mixed Reality (MxR) scenario that can be applied to VH applications that aim at enhancing cultural learning in situ

Spatiotemporal information extraction from a historic expedition gazetteer

Historic expeditions are events that are flavored by exploratory, scientific, military or geographic characteristics. Such events are often documented in literature, journey notes or personal diaries. A typical historic expedition involves multiple site visits and their descriptions contain spatiotemporal and attributive contexts. Expeditions involve movements in space that can be represented by triplet features (location, time and description). However, such features are implicit and innate parts of textual documents. Extracting the geospatial information from these documents requires understanding the contextualized entities in the text. To this end, we developed a semi-automated framework that has multiple Information Retrieval and Natural Language Processing components to extract the spatiotemporal information from a two-volume historic expedition gazetteer. Our framework has three basic components, namely, the Text Preprocessor, the Gazetteer Processing Machine and the JAPE (Java Annotation Pattern Engine) Transducer. We used the Brazilian Ornithological Gazetteer as an experimental dataset and extracted the spatial and temporal entities from entries that refer to three expeditioners’ site visits (which took place between 1910 and 1926) and mapped the trajectory of each expedition using the extracted information. Finally, one of the mapped trajectories was manually compared with a historical reference map of that expedition to assess the reliability of our framework

Walkable Mixed Reality Map as interaction interface for Virtual Heritage

© 2019 Elsevier Ltd. Studies in the Virtual Heritage (VH) domain have led to underlining the significance of a contextual relationship between users, immersive reality technologies and interactive and engaging cultural context as essential aspects towards enabling cultural learning in VH applications. Interaction methods, therefore, play a great role in terms of enabling interactive and engaging experience under various VH settings. This paper proposes a novel approach to use maps as interaction interfaces in a Mixed Reality (MxR) environment that could be applied to specific VH settings with a predefined cultural and historical context. The primary focus of the proposed interaction method named ‘Walkable MxR Map’ is to use interactive, immersive and walkable maps to allow users interact with cultural content, 3D models, and different multimedia content at museums and heritage sites. Hence, the applicability of the proposed systems will not be limited to museums' indoor settings; its applicability extends outdoors at the natural location of cultural heritage assets. To this end, immersive reality technologies, interaction methods, development platforms and mapping and cloud storage services have been combined to realise the interaction method. The Walkable MxR Map allows users to interact with virtual objects via maps that are virtually projected on the floor and viewable through MxR devices, specifically the Microsoft HoloLens. The projected maps are room-scale and walkable with a potential global scalability. Besides movement-based interaction, users can interact with virtual objects, multimedia content and 3D models using HoloLens's standard gesture, gaze and voice interaction methods

Clouds-Based Collaborative and Multi-Modal Mixed Reality for Virtual Heritage

Recent technological advancements in immersive reality technologies have become a focus area in the virtual heritage (VH) domain. In this regard, this paper attempts to design and implement clouds-based collaborative and multi-modal MR application aiming at enhancing cultural learning in VH. The design and implementation can be adopted by the VH domain for various application themes. The application utilises cloud computing and immersive reality technologies. The use of cloud computing, collaborative, and multi-modal interaction methods is influenced by the following three issues. First, studies show that users’ interaction with immersive reality technologies and virtual environments determines their learning outcome and the overall experience. Second, studies also demonstrate that collaborative and multi-modal interaction methods enable engagement in immersive reality environments. Third, the integration of immersive reality technologies with traditional museums and cultural heritage sites is getting significant attention in the domain. However, a robust approach, development platforms (frameworks) and easily adopted design and implementation approaches, or guidelines are not commonly available to the VH community. This paper, therefore, will attempt to achieve two major goals. First, it attempts to design and implement a novel application that integrates cloud computing, immersive reality technology and VH. Second, it attempts to apply the proposed application to enhance cultural learning. From the perspective of cultural learning and users’ experience, the assumption is that the proposed approach (clouds-based collaborative and multi-modal MR) can enhance cultural learning by (1) establishing a contextual relationship and engagement between users, virtual environments and cultural context in museums and heritage sites, and (2) by enabling collaboration between users

Redefining Mixed Reality: User-Reality-Virtuality and Virtual Heritage Perspectives

The primary objective of this paper is to present a redefinition of Mixed Reality from a perspective emphasizing the relationship between users, virtuality and reality as a fundamental component. The redefinition is motivated by three primary reasons. Firstly, current literature in which Augmented Reality is the focus appears to approach Augmented Reality as an alternative to Mixed Reality. Secondly, Mixed Reality is often considered to encompass Augmented Reality and Virtual Reality rather than specifying it as a segment along the reality-virtuality continuum. Thirdly, most common definitions of Augmented Reality (AR), Augmented Virtuality (AV), Virtual Reality (VR) and Mixed Reality (MxR) in current literature are based on outdated display technologies, and a relationship between virtuality and reality, neglecting the importance of the users necessarily complicit sense of immersion from the relationship. The focus of existing definitions is thus currently technological, rather than experiential. We resolve this by redefining the continuum and MxR, taking into consideration the experiential symbiotic relationship and interaction between users, reality, and current virtual reality technologies. In addition, the paper will suggest some high-level overview of the redefinition’s contextual applicability to the Virtual Heritage (VH) domain

From photo to 3D to mixed reality: A complete workflow for cultural heritage visualisation and experience

The domain of cultural heritage is on the verge of adopting immersive technologies; not only to enhance user experience and interpretation but also to satisfy the more enthusiastic and tech-savvy visitors and audiences. However, contemporary academic discourse seldom provides any clearly defined and versatile workflows for digitising 3D assets from photographs and deploying them to a scalable 3D mixed reality (MxR)environment; especially considering non-experts with limited budgets. In this paper, a collection of open access and proprietary software and services are identified and combined via a practical workflow which can be used for 3D reconstruction to MxR visualisation of cultural heritage assets. Practical implementations of the methodology has been substantiated through workshops and participants’ feedback. This paper aims to be helpful to non-expert but enthusiastic users (and the GLAM sector)to produce image-based 3D models, share them online, and allow audiences to experience 3D content in a MxR environment

The Influence of Collaborative and Multi-Modal Mixed Reality: Cultural Learning in Virtual Heritage

Studies in the virtual heritage (VH) domain identify collaboration (social interaction), engagement, and a contextual relationship as key elements of interaction design that influence users’ experience and cultural learning in VH applications. The purpose of this study is to validate whether collaboration (social interaction), engaging experience, and a contextual relationship enhance cultural learning in a collaborative and multi-modal mixed reality (MR) heritage environment. To this end, we have designed and implemented a cloud-based collaborative and multi-modal MR application aiming at enhancing user experience and cultural learning in museums. A conceptual model was proposed based on collaboration, engagement, and relationship in the context of MR experience. The MR application was then evaluated at the Western Australian Shipwrecks Museum by experts, archaeologists, and curators from the gallery and the Western Australian Museum. Questionnaire, semi-structured interview, and observation were used to collect data. The results suggest that integrating collaborative and multi-modal interaction methods with MR technology facilitates enhanced cultural learning in VH

A survey of augmented, virtual, and mixed reality for cultural heritage

© 2018 ACM. A multimedia approach to the diffusion, communication, and exploitation of Cultural Heritage (CH) is a well-established trend worldwide. Several studies demonstrate that the use of new and combined media enhances how culture is experienced. The beneft is in terms of both number of people who can have access to knowledge and the quality of the diffusion of the knowledge itself. In this regard, CH uses augmented-, virtual-, and mixed-reality technologies for different purposes, including education, exhibition enhancement, exploration, reconstruction, and virtual museums. These technologies enable user-centred presentation and make cultural heritage digitally accessible, especially when physical access is constrained. A number of surveys of these emerging technologies have been conducted; however, they are either not domain specific or lack a holistic perspective in that they do not cover all the aspects of the technology. A review of these technologies from a cultural heritage perspective is therefore warranted. Accordingly, our article surveys the state-of-the-art in augmented-, virtual-, and mixed-reality systems as a whole and from a cultural heritage perspective. In addition, we identify specific application areas in digital cultural heritage and make suggestions as to which technology is most appropriate in each case. Finally, the article predicts future research directions for augmented and virtual reality, with a particular focus on interaction interfaces and explores the implications for the cultural heritage domain